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* ---^ T-his is the fifth and final paper of a^symposiui& on 

the evaluation of juvenile divei;sion programs by the Claremont 
Graduate School C&irter for Applied Social Research* Xhe first four . 
papers deal wi-th specific methodologies tfsed to obtain information at 
each of four levels of evaluei^tion: general project management, 
description, of tthe intervention, assessment of client outcomes, and 
analysis of systeiD^wide impact. This fifth paper provides an overview 
of how the 'specific methodologies fit into^ the framework o'f our 
Qvexall evaluation plan. A key concept in the author's approach to 
evaluation is the use of multiple sources of information to address^ 
each major program issue (e*g. , Vhat is the impact of diversion ^ 
programs* on .the juvenile ju^stice system?) Thus, a number of data 
sources in addition to those descrit)ed in the methodological papers : 
were used, including archival sources, questionnaires, ai>d 
interviews. This paper includes a discussion of t&ese aspects of 'the 

.^valuation effort, ^nd focuses on relationships between various' 
methodological approaches* Another topic discussed is the difficulty 

^evalua tors face in attempting sxmultaneou^^ly to falfill . 
respdnsibilities to diverse constituencies iiK:luding clients, project 
staff, service proviifers, the community, and funding agencies and 
political bodies, (Author) * ;x . ■ ■ * ' 
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So far the papers in this feymposium have been addressed to particular , 
methodologies that we h^\rfe used in the course of bur evaluati-Gns . I wouj-d 
like to talk about the factors that are Involved in the decision to use the 

■ ' ■ ' . •.■ . . - o 

.various technique^ for dqta collection.' Perhaps bhe obvious prcoedure would, 

be to identify the 'goals -of ' th& program, settle on priorities, artd then pro- 
■ , ' " ' ' « ' (. - ' 
ceed to gather data to determine whether the goals have been attained. How- 

ever, things are not- so simple. - ■ ■ ^ 

The first pro'blem is" that the^evaluatlon effort has diverse constit- 

• u4ncies, each with' its -ow^ goals', and its own pifiorlties on the issues to be 
evaluated* In particular, there dre funding agencies, management b^ -.pds, ' ^ 
project staffs, clients^and' the coinmunilzy at large, plus other agencies 
that may be affected by the program such as police det)&rtntents, the proba- 
tion department, and community agehcles; that provide ser^-ices to youth. Now 
the intersts of all of these groups do not necessarily 'coincide , and in many', 
instances there are areas of competition or antago^^ism^ Each of these groups, 
except perhaps the clients and the community, are in a position to exert at 

.least -some-political pressure, on each 'other and either directly or indi- 
rectly, on the evaluators as well. Any data that the evaluators report or 
conclusions that they draw wil^ be intetprg^ed by each ^o-P ^"^^^ its ovm^ 
point of view apd for irs purpose*. Thus, aji evaluation report that 



includes data on the operation of a sysjieta aijd raakes recortraiendations for 
change, is ditxned ta be' the focal point of political*battles, and in any ^ 



tUBe is ^ likely to be treated with suaplulon in some quarterB,^ and perhaps) 
even be facod with a lack of ^cooperation. ' . ,) 

* I'd like to digress for a minute to* t:/l\t about one othe^ constituency 
of Id he evaluation effort in our operatfbr^, and that-^is.qur graduate stucients 
and ourt. education- and training program. Our evalu^ion team operates ui^der 
'j^he lOTbrella of a recently formed Center ^or A'pplied Social Research at 

: • 7^. ) ' - — ^ * • ■ 

QJaremont Graduate S/hool. One goal of the research center is tp provide to^ 

graduate students tlje *opportunity for a range of/practical experiences in h- 

^ ' . ^ 7 . ^ ' , / ' ' . / r . 

apolylng methods and theotjr from psychology to diverse settings. Students 
yill ordinarily be part of a. team headed by advanced ^raduate;students under 
factulty direction. Students with limited experience start with routine 
jobs such as coding, but have' the opppr^^unity to mpv^ into 'supervisory roles ^ 
as they ^gain experience.. Several students have taken on specific aspects " 
of the ^valuatipn effort and are worTcing 'on dissertations in.ponjunation 
with the evalliatioTl. The advanced graduate students play a m^jor role in* 
designing research procedures, writing reports and proposals, and repre- 
senting the evaluation teaman the field* Thus tlie evaluation effort can 
provide* a vehicle for training graduate stude^its in evaluation methods and 
procedures, and thereby meet needs of a graduate program witl^.- an 'emphasis 
-on Public Affair? Psychdlogy. ' , 

^* : ^ I mentioned earlier, the evaluation effort also must address, the 
demands of a diverge coniJtituency. Wei might look at some of jihese in more 
detail. First, 'there, are the funding agencies whichf need information on* 
^w^ch to ^ase"^ decisior^ for continuation of funding' and for changing guide- 

lines. They need to kpow whether their ihoney *is being ^ell spent. In par- 

/--..*■' * ' . " . ^ 

ticular,^has the program attaineH the go"als stated in' the propos^, and is 



id operating in a reasonably cost-effective manner. A second groufi is the 
management boards who'-ncetl information to gui^de policy-making; aiich as de- 

fining -criterj-a for -referral to the firogriim, and to identify probXfjm areas 

y f>/ . ■ ■ 

in the operation of the program. ' " ■ ^ 

- ■ ; ^ 

. A third group is the project staff who need^ information at a level of 

greater detail. They want to be able to document -achievements of the pro- 

gram, monitor operations, land p>inpoint problems ^t a very specific- level. 

m * ' * * 

Foj: example, how^ effective are individual .service providers, how should 

' . * * ' * * 

clients be matched tfd service, should service'he limited to, sdy,,10 hours - 

or 15 hours? A program like juvenile diversion also has an i^npact on three 

other broad groups. The first of tlies^ are the various other agencies that 

deal with ybuth, such as thtf pre* ition departm^t ; various agencies that 

• - ■ . ; . ^. , 

provide youth servrees, -and the schools- ♦For the points of view of these 

agencies, the c^iversion program can be seen either as a threat or. as a re- 

source* Another interested "groi^p would* be the general community, people who 

'are affected by the behavior of juveniles in society- The^last group, b^it 

hopefully not tlie least, would be the"^juveniles themselves and their fain- 

ilies. . ^ ■ * 

,dbv4.pusly, an evaluator neieds to* set ■prior:^ties among the issues to * 

be*addressed by the evaluation. The naturals starting point would be the. 

- y 

' evaluation contract and tHe statement of goals in the proposal which was ^ 
funded. However, It would not be a, good procedure to adopt metho*dologies 
that have no flexibility to respond to changes in the program. For- example* 

one of the major goafsTof =^tjie Orange .County Regional Diversion Project wljen 

^' * " . * ' . *\ ^ ' . ' * * 

it was founded in 1976 w^S to Eliminate pentratlon of status offenders into 

* ■ - ' 

the Pjr<5}>ation . Department* However ,^ on" January 1* 1977 a state lay (Assembly 



Bill v^21) took tifftjctr, which aevcrtfsly llniitLKt the uoe of conf inemtsnt £or 

status 'offcirdeVs. Later on In the year, the prc:jldlng Juvonilc Court JudgQ ' 

in drange County issued an order that Probation DepartAient would no 

longer be allowed to accept any status offenders except chronic truants, * 

Thus, a time sejrlcs'^analysis of intake of .status, offenders to fhe Probation 

Department would shpw ;that one goal of the Diver^ipn Program had been met in 
* ' * * ■ * < * * / ^ 

a most dramatic way — penetration of status offenders into the Probation Depart 

v.- • ■ ■ ■ ■ ■ 

ment had been eliminated in one ^ear! Of course, the true issue had shifted 

to the question of whg.t was now happening to youth who fcarlier would have. 

been referred to Probation for staCu's offenses, and how the diversion pro-- . , 

gCam cpuld b^st seiive them* ^ . ^ 

Another goal of th^ diversion program is to take youtji who would be 

referred to the Probation Department because they had .commitl;ed a penal 

of fense,^' and instead Miv.ert* them into counseling or ot;her youth service 
* 

programs- A problem faced by the evaluators is to determine which' j^veni^es' 
wo ul 4 have gone into Probation if they had not been diverted^ or at least 
form an' estimate of that number* ^ There is, of <:ourse, no , way to provide a ' 
certain answer to the question of what would have happened* There is na 
single data source that is free from extraneous influences* The*logic re- 

minds me of the *Sweet Sixteen" speaker system. Shortly after WorlH War 

■ ~t * ■ * " * 

II the ^apagese produced very* cheap handmade speakers as a cottage industry: . 
The speakers had all sorts of individual iinperfections, and^ they, sounded 
terrible* Some unsung hero had ^the ingfeAuity to put together a 4x4 ma?rix 
of these little speakers, thereby creating the 'Sweet Sixteen V speaker , 
which actually had^quite^ good qual ft Il*s% ^ - * , - ' 

In , addition to'developing alternate, measures for the same question, we 



have found "it nccoaaaty to t'otmulate^ivl Lci nal o c]ucr>tloria to acVltcsn some 

\ , ^ ^ ' ■ ' ■ 

general iasues. Co'nsldeV, Eor cxaqiplc jfr:he prablctii of nncasurlng tlic ligpact: 

■ * ^ * " ' ' # '* 

"of diyctsWn services on "juveniles, An thc^papar on evaluation' of clini^^al 

process, Mike, Sheryl, , and Jim have pointed out the limitations of recidi- 

vism rates as the sole outcome mea34ire,* Hp have foUnd it useful to dietln- 

.guish three groups of clients for wljicli one. should, logically^ use "different 

outcofae measures. First, there* are the true 'diversions' frpm law enforce- , 

ment of juveniles who would otherwise have been sent tosfhe Probation D^part-r 

ment. A second group includes those who have violated the law, but would 
rf*^ ■ • ' * ' * 

' ■ . - 

not have been sent to the ■ Probation De*partinenD this time. J^or thesi^ 

■ • * ■ ■■■.'...>, ■ - • . . 

youth, referral to a diversion project is actually an effort to prevent fift- 

■ ' " . • • ... 
ther trouble witih'thfe Isfw.* For both o^.thse groups, recidlvistic law en-^^j 

forcement contact is a useful Index o^, the effectiveness of diversioa, al- 

though the groups Taigh"t be expected to differ in- severity of recidlvistic 

offenses. ,A third group Qohsists^of those youths who have nat violated the- 

law, but who are referred for counsellfig because "someone feels they have a 

need f^r (tielp-. For thes6 youth, law enforcj^ment recidivism is rtD't aa'appro- 

priate index, since they-^are not necessarily expected to ever have contact 

with law enforcement. A njjmber of Supplementary measures. are needed^ such , 

as school grades and behavior, ratings, by Counselors, parents, and*the cli- 

ents them<;elve3. Thus, the set of -outcome measures used to assess impact 

of diversi^on services differs according to the nature of the-cases. * 

^ ' . ' / 

*^^n sum, we have found that we have had to use "a^tremendous variety of 

r . ' t * 

data. We have had to StriK^ a balance betve.en. collecting all the dat^ we 



would like and collectitig only the most basic information- _^nother bapance 



we have tried to maintain is- with, the people who_ are the consumers of our 
evaluation effort. On the one hand, we have developed a close working 

i ■ • ' . . 



XI . 



J ■ 



rclationsfiiv! wliUc- on the other we hMVc^ecn careful to protect our inde- 
pendeAcc and our objccclvity. In the spirit of balfiHcluK v:Lcwiiointa» wc 
will pow turn dho progr£lm ovec ^to some oE the people who ace' involved in 
running diyersiort proijrama* ^IfS so are among the /coheumers* oC out evalu 
atlon research* , • * " - ,. 
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